Large-Scale Estimation and Analysis of Web Users' Mood from Web Search Query and Mobile Sensor Data.

The ability to estimate the current mood states of web users has considerable potential for realizing user-centric opportune services in pervasive computing. However, it is difficult to determine the data type used for such estimation and collect the ground truth of such mood states. Therefore, we built a model to estimate the mood states from search-query data in an easy-to-collect and non-invasive manner. Then, we built a model to estimate mood states from mobile sensor data as another estimation model and supplemented its output to the ground-truth label of the model estimated from search queries. This novel two-step model building contributed to boosting the performance of estimating the mood states of web users. Our system was also deployed in the commercial stack, and large-scale data analysis with >11 million users was conducted. We proposed a nationwide mood score, which bundles the mood values of users across the country. It shows the daily and weekly rhythm of people's moods and explains the ups and downs of moods during the COVID-19 pandemic, which is inversely synchronized to the number of new COVID-19 cases. It detects big news that simultaneously affects the mood states of many users, even under fine-grained time resolution, such as the order of hours. In addition, we identified a certain class of advertisements that indicated a clear tendency in the mood of the users who clicked such advertisements.

Associations of active and passive smartphone use with measures of youth mental health during the COVID-19 pandemic.

Smartphone use provides a significant amount of screen-time for youth, and there have been growing concerns regarding its impact on their mental health. While time spent in a passive manner on the device is frequently considered deleterious, more active engagement with the phone might be protective for mental health. Recent developments in mobile sensing technology provide a unique opportunity to examine behaviour in a naturalistic manner. The present study sought to investigate, in a sample of 451 individuals (mean age 20.97 years old, 83% female), whether the amount of time spent on the device, an indicator of passive smartphone use, would be associated with worse mental health in youth and whether an active form of smartphone use, namely frequent checking of the device, would be associated with better outcomes. The findings highlight that overall time spent on the smartphone was associated with more pronounced internalizing and externalizing symptoms in youth, while the number of unlocks was associated with fewer internalizing symptoms. For externalizing symptoms, there was also a significant interaction between the two types of smartphone use observed. Using objective measures, our results suggest interventions targeting passive smartphone use may contribute to improving the mental health of youth.

Leveraging Mobile Sensing and Bayesian Change Point Analysis to Monitor Community-scale Behavioral Interventions: A Case Study on COVID-19

During pandemics, effective interventions require monitoring the problem at different scales and understanding the various tradeoffs between efficacy, privacy, and economic burden. To address these challenges, we propose a framework where we perform Bayesian change-point analysis on aggregate behavior markers extracted from mobile sensing data collected during the COVID-19 pandemic. Results generated by 598 participants for up to four months reveal rich insights: We observe an increase in smartphone usage around February 10th, followed by an increase in email usage around February 27th and, finally, a large reduction in participant's mobility around March 13th. These behavior changes overlapped with important news events and government directives such as the naming of COVID-19, a spike in the number of reported cases in Europe, and the declaration of national emergency by President Trump. We also show that our detected change points align with changes in large scale external sources, including number of COVID-19 tweets, COVID-19 search traffic, and a large-scale foot traffic data collected by SafeGraph, providing further validation of our method. Our results show promise towards the feasibility of using mobile sensing to understand communities' responses to public health interventions. © 2022 Copyright held by the owner/author(s).

Nighttime Continuous Contactless Smartphone-Based Cough Monitoring for the Ward: Validation Study.

BACKGROUND: Clinical deterioration can go unnoticed in hospital wards for hours. Mobile technologies such as wearables and smartphones enable automated, continuous, noninvasive ward monitoring and allow the detection of subtle changes in vital signs. Cough can be effectively monitored through mobile technologies in the ward, as it is not only a symptom of prevalent respiratory diseases such as asthma, lung cancer, and COVID-19 but also a predictor of acute health deterioration. In past decades, many efforts have been made to develop an automatic cough counting tool. To date, however, there is neither a standardized, sufficiently validated method nor a scalable cough monitor that can be deployed on a consumer-centric device that reports cough counts continuously. These shortcomings limit the tracking of coughing and, consequently, hinder the monitoring of disease progression in prevalent respiratory diseases such as asthma, chronic obstructive pulmonary disease, and COVID-19 in the ward. OBJECTIVE: This exploratory study involved the validation of an automated smartphone-based monitoring system for continuous cough counting in 2 different modes in the ward. Unlike previous studies that focused on evaluating cough detection models on unseen data, the focus of this work is to validate a holistic smartphone-based cough detection system operating in near real time. METHODS: Automated cough counts were measured consistently on devices and on computers and compared with cough and noncough sounds counted manually over 8-hour long nocturnal recordings in 9 patients with pneumonia in the ward. The proposed cough detection system consists primarily of an Android app running on a smartphone that detects coughs and records sounds and secondarily of a backend that continuously receives the cough detection information and displays the hourly cough counts. Cough detection is based on an ensemble convolutional neural network developed and trained on asthmatic cough data. RESULTS: In this validation study, a total of 72 hours of recordings from 9 participants with pneumonia, 4 of whom were infected with SARS-CoV-2, were analyzed. All the recordings were subjected to manual analysis by 2 blinded raters. The proposed system yielded a sensitivity and specificity of 72% and 99% on the device and 82% and 99% on the computer, respectively, for detecting coughs. The mean differences between the automated and human rater cough counts were -1.0 (95% CI -12.3 to 10.2) and -0.9 (95% CI -6.5 to 4.8) coughs per hour within subject for the on-device and on-computer modes, respectively. CONCLUSIONS: The proposed system thus represents a smartphone cough counter that can be used for continuous hourly assessment of cough frequency in the ward.

Predicting Multiple Sclerosis Outcomes During the COVID-19 Stay-at-home Period: Observational Study Using Passively Sensed Behaviors and Digital Phenotyping.

BACKGROUND: The COVID-19 pandemic has broad negative impact on the physical and mental health of people with chronic neurological disorders such as multiple sclerosis (MS). OBJECTIVE: We presented a machine learning approach leveraging passive sensor data from smartphones and fitness trackers of people with MS to predict their health outcomes in a natural experiment during a state-mandated stay-at-home period due to a global pandemic. METHODS: First, we extracted features that capture behavior changes due to the stay-at-home order. Then, we adapted and applied an existing algorithm to these behavior-change features to predict the presence of depression, high global MS symptom burden, severe fatigue, and poor sleep quality during the stay-at-home period. RESULTS: Using data collected between November 2019 and May 2020, the algorithm detected depression with an accuracy of 82.5% (65% improvement over baseline; F1-score: 0.84), high global MS symptom burden with an accuracy of 90% (39% improvement over baseline; F1-score: 0.93), severe fatigue with an accuracy of 75.5% (22% improvement over baseline; F1-score: 0.80), and poor sleep quality with an accuracy of 84% (28% improvement over baseline; F1-score: 0.84). CONCLUSIONS: Our approach could help clinicians better triage patients with MS and potentially other chronic neurological disorders for interventions and aid patient self-monitoring in their own environment, particularly during extraordinarily stressful circumstances such as pandemics, which would cause drastic behavior changes.

Enhancing Self-protection: What Influences Human’s Epidemic Prevention Behavior during the COVID-19 Pandemic

Under the circumstance of the rapid spread of the COVID-19 pandemic, enhancing human’s awareness of self-protection is one practical method to slow down the epidemic. In this study, we utilize mobile sensing to track human activity and guide human’s epidemic prevention behavior by gamified feedback techniques by our developed application. Virtually, human’s self-protection awareness is affected by many factors and the measures to enhance people’s self-protection behavior against the epidemic COVID-19 has always been an unresolved issue. In order to search for factors that influence human’s self-protection behavior, we analyzed the relationships between various human activities and the percentage complete of human’s self-protection behavior and we have extracted some more general conclusions from the results. Based on our data analysis results, we also made some proposals to enhance self-protection behavior. Meanwhile, our study illustrates the effectiveness of the method that analyzes human self-protection behavior through mobile sensing. Our study also validates the effectiveness of persuasive technology on human’s self-protection behavior against the COVID-19 pandemic and therefore we advocate enhancing human’s self-protection awareness through external intervention and guidance by smart device. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

COVID Student Study: A Year in the Life of College Students during the COVID-19 Pandemic Through the Lens of Mobile Phone Sensing

The COVID-19 pandemic continues to affect the daily life of college students, impacting their social life, education, stress levels and overall mental well-being. We study and assess behavioral changes of N=180 undergraduate college students one year prior to the pandemic as a baseline and then during the first year of the pandemic using mobile phone sensing and behavioral inference. We observe that certain groups of students experience the pandemic very differently. Furthermore, we explore the association of self-reported COVID-19 concern with students' behavior and mental health. We find that heightened COVID-19 concern is correlated with increased depression, anxiety and stress. We evaluate the performance of different deep learning models to classify student COVID-19 concerns with an AUROC and F1 score of 0.70 and 0.71, respectively. Our study spans a two-year period and provides a number of important insights into the life of college students during this period. © 2022 Owner/Author.

From Personalized Medicine to Population Health: A Survey of mHealth Sensing Techniques

Mobile sensing systems have been widely used as a practical approach to collect behavioral and health-related information from individuals and to provide timely intervention to promote health and well-being, such as mental health and chronic care. As the objectives of mobile sensing could be either personalized medicine for individuals or public health for populations, in this work we review the design of these mobile sensing systems, and propose to categorize the design of these systems in two paradigms –(i) Personal Sensing and (ii) Crowd Sensing paradigms. While both sensing paradigms might incorporate common ubiquitous sensing technologies, such as wearable sensors, mobility monitoring, mobile data offloading, and cloud-based data analytics to collect and process sensing data from individuals, we present two novel taxonomy systems based on the (a) Sensing Objectives (e.g., goals of mHealth sensing systems and how technologies achieve the goals), and (b) the Sensing Systems Design and Implementation (D&I) (e.g., designs of mHealth sensing systems and how technologies are implemented). With respect to the two paradigms and two taxonomy systems, this work systematically reviews this field. Specifically, we first present technical reviews on the mHealth sensing systems in eight common/popular healthcare issues, ranging from depression and anxiety to COVID-19. Through summarizing the mHealth sensing systems, we comprehensively survey the research works using the two taxonomy systems, where we systematically review the Sensing Objectives and Sensing Systems D&I while mapping the related research works onto the life-cycles of mHealth Sensing, i.e., (1) Sensing Task Creation &Participation, (2) Health Surveillance &Data Collection, and (3) Data Analysis &Knowledge Discovery. In addition to summarization, the proposed taxonomy systems also help the potential directions of mobile sensing for health from both personalized medicine and population health perspectives. Finally, we attempt to test and discuss the validity of our scientific approaches to the survey. IEEE

Detection and Analysis of Interrupted Behaviors by Public Policy Interventions during COVID-19

In most countries around the world, various public policies and guidelines, such as social distancing and stay-at-home orders, have been put in place to slow down the spreading of COVID-19. Relying on traditional surveys to assess policy impacts on community level behavior changes may lead to biased results, and limit fine-grained understanding of human behavior dynamics over time. We propose to leverage mobile sensing to capture people's behavior footprints amid the COVID-19 pandemic, and understand their collective behavior changes with respect to existing policies. Specifically, we propose to extract a rich set of behavioral markers from raw mobile sensing data, including mobility, social interactions, physical activities, and health states, and apply them in a generalized behavior change analysis framework to measure and detect community level behavior changes in an epidemic context. We present how to combine change point detection algorithm and interrupted time series analysis to automatically detect three different measurements of behavior changes (e.g., level, trend, and variance changes), and provide insights supported by statistical inference. A case study using a dataset that we collected from a large mobile sensing study conducted in the United States is shown to demonstrate the proposed framework and method. © 2021 IEEE.

DEEP PEDESTRIAN DENSITY ESTIMATION FOR SMART CITY MONITORING

Recently, requirement of city monitoring and maintenance using ICT techniques increases with the help of transportation system. In addition, the spread of COVID-19 has increased the demand for managing pedestrian traffic volume. To contribute to these trends, in this paper, we propose a new pedestrian radar map system in order to estimate pedestrian density on streets and sidewalks. Our system uses e-bikes to collect 360-degree images and visualize pedestrian positions as a radar map. In evaluations, we confirm the accuracies of the radar maps and pedestrian density by using KITTI dataset and by carrying out a field experiment. © 2021 IEEE

Variability in Physical Inactivity Responses of University Students during COVID-19 Pandemic: A Monitoring of Daily Step Counts Using a Smartphone Application.

This study investigated the changes in physical inactivity of university students during the COVID-19 pandemic, with reference to their academic calendar. We used the daily step counts recorded by a smartphone application (iPhone Health App) from April 2020 to January 2021 (287 days) for 603 participants. The data for 287 days were divided into five periods based on their academic calendar. The median value of daily step counts across each period was calculated. A k-means clustering analysis was performed to classify the 603 participants into subgroups to demonstrate the variability in the physical inactivity responses. The median daily step counts, with a 7-day moving average, dramatically decreased from 5000 to 2000 steps/day in early April. It remained at a lower level (less than 2000 steps/day) during the first semester, then increased to more than 5000 steps/day at the start of summer vacation. The clustering analysis demonstrated the variability in physical inactivity responses. The inactive students did not recover daily step counts throughout the year. Consequently, promoting physical activity is recommended for inactive university students over the course of the whole semester.

Autonomous Multi-Rotor Aerial Platform for Air Pollution Monitoring.

During the last few years, scientists have become increasingly concerned about air quality. Particularly in large cities and industrialised areas, air quality is affected by pollution from natural and anthropogenic sources and this has a significant impact on human health. Continuous monitoring of air quality is an important step in investigating the causes and reducing pollution. In this paper, we propose a new autonomous multi-rotor aerial platform that can be used to perform real-time monitoring of air quality in large cities. The air quality monitoring system is able to cover large areas, with high spatial resolution, even above average buildings, while being relatively low cost. We evaluate the proposed system in several locations throughout a metropolitan city, during different seasons and generate fine-grained heat-maps that display the level of pollution of specific areas based on different altitudes.

On the Transition of Social Interaction from In-Person to Online: Predicting Changes in Social Media Usage of College Students during the COVID-19 Pandemic based on Pre-COVID-19 On-Campus Colocation.

Pandemics significantly impact human daily life. People throughout the world adhere to safety protocols (e.g., social distancing and self-quarantining). As a result, they willingly keep distance from workplace, friends and even family. In such circumstances, in-person social interactions may be substituted with virtual ones via online channels, such as, Instagram and Snapchat. To get insights into this phenomenon, we study a group of undergraduate students before and after the start of COVID-19 pandemic. Specifically, we track N=102 undergraduate students on a small college campus prior to the pandemic using mobile sensing from phones and assign semantic labels to each location they visit on campus where they study, socialize and live. By leveraging their colocation network at these various semantically labeled places on campus, we find that colocations at certain places that possibly proxy higher in-person social interactions (e.g., dormitories, gyms and Greek houses) show significant predictive capability in identifying the individuals' change in social media usage during the pandemic period. We show that we can predict student's change in social media usage during COVID-19 with an F1 score of 0.73 purely from the in-person colocation data generated prior to the pandemic.

Mobile Sensing with Smart Wearables of the Physical Context of Distance Learning Students to Consider Its Effects on Learning.

Research shows that various contextual factors can have an impact on learning. Some of these factors can originate from the physical learning environment (PLE) in this regard. When learning from home, learners have to organize their PLE by themselves. This paper is concerned with identifying, measuring, and collecting factors from the PLE that may affect learning using mobile sensing. More specifically, this paper first investigates which factors from the PLE can affect distance learning. The results identify nine types of factors from the PLE associated with cognitive, physiological, and affective effects on learning. Subsequently, this paper examines which instruments can be used to measure the investigated factors. The results highlight several methods involving smart wearables (SWs) to measure these factors from PLEs successfully. Third, this paper explores how software infrastructure can be designed to measure, collect, and process the identified multimodal data from and about the PLE by utilizing mobile sensing. The design and implementation of the Edutex software infrastructure described in this paper will enable learning analytics stakeholders to use data from and about the learners' physical contexts. Edutex achieves this by utilizing sensor data from smartphones and smartwatches, in addition to response data from experience samples and questionnaires from learners' smartwatches. Finally, this paper evaluates to what extent the developed infrastructure can provide relevant information about the learning context in a field study with 10 participants. The evaluation demonstrates how the software infrastructure can contextualize multimodal sensor data, such as lighting, ambient noise, and location, with user responses in a reliable, efficient, and protected manner.

Mental Health and Behavior of College Students During the COVID-19 Pandemic: Longitudinal Mobile Smartphone and Ecological Momentary Assessment Study, Part II.

BACKGROUND: Since late 2019, the lives of people across the globe have been disrupted by COVID-19. Millions of people have become infected with the disease, while billions of people have been continually asked or required by local and national governments to change their behavioral patterns. Previous research on the COVID-19 pandemic suggests that it is associated with large-scale behavioral and mental health changes; however, few studies have been able to track these changes with frequent, near real-time sampling or compare these changes to previous years of data for the same individuals. OBJECTIVE: By combining mobile phone sensing and self-reported mental health data in a cohort of college-aged students enrolled in a longitudinal study, we seek to understand the behavioral and mental health impacts associated with the COVID-19 pandemic, measured by interest across the United States in the search terms coronavirus and COVID fatigue. METHODS: Behaviors such as the number of locations visited, distance traveled, duration of phone use, number of phone unlocks, sleep duration, and sedentary time were measured using the StudentLife mobile smartphone sensing app. Depression and anxiety were assessed using weekly self-reported ecological momentary assessments, including the Patient Health Questionnaire-4. The participants were 217 undergraduate students. Differences in behaviors and self-reported mental health collected during the Spring 2020 term, as compared to previous terms in the same cohort, were modeled using mixed linear models. RESULTS: Linear mixed models demonstrated differences in phone use, sleep, sedentary time and number of locations visited associated with the COVID-19 pandemic. In further models, these behaviors were strongly associated with increased interest in COVID fatigue. When mental health metrics (eg, depression and anxiety) were added to the previous measures (week of term, number of locations visited, phone use, sedentary time), both anxiety and depression (P<.001) were significantly associated with interest in COVID fatigue. Notably, these behavioral and mental health changes are consistent with those observed around the initial implementation of COVID-19 lockdowns in the spring of 2020. CONCLUSIONS: In the initial lockdown phase of the COVID-19 pandemic, people spent more time on their phones, were more sedentary, visited fewer locations, and exhibited increased symptoms of anxiety and depression. As the pandemic persisted through the spring, people continued to exhibit very similar changes in both mental health and behaviors. Although these large-scale shifts in mental health and behaviors are unsurprising, understanding them is critical in disrupting the negative consequences to mental health during the ongoing pandemic.

Mobile Sensing Apps and Self-management of Mental Health During the COVID-19 Pandemic: Web-Based Survey.

BACKGROUND: During the COVID-19 pandemic, people had to adapt their daily life routines to the currently implemented public health measures, which is likely to have resulted in a lack of in-person social interactions, physical activity, or sleep. Such changes can have a significant impact on mental health. Mobile sensing apps can passively record the daily life routines of people, thus making them aware of maladaptive behavioral adjustments to the pandemic. OBJECTIVE: This study aimed to explore the views of people on mobile sensing apps that passively record behaviors and their potential to increase awareness and helpfulness for self-managing mental health during the pandemic. METHODS: We conducted an anonymous web-based survey including people with and those without mental disorders, asking them to rate the helpfulness of mobile sensing apps for the self-management of mental health during the COVID-19 pandemic. The survey was conducted in May 2020. RESULTS: The majority of participants, particularly those with a mental disorder (n=106/148, 72%), perceived mobile sensing apps as very or extremely helpful for managing their mental health by becoming aware of maladaptive behaviors. The perceived helpfulness of mobile sensing apps was also higher among people who experienced a stronger health impact of the COVID-19 pandemic (ß=.24; 95% CI 0.16-0.33; P<.001), had a better understanding of technology (ß=.17; 95% CI 0.08-0.25; P<.001), and had a higher education (ß=.1; 95% CI 0.02-0.19; P=.02). CONCLUSIONS: Our findings highlight the potential of mobile sensing apps to assist in mental health care during the pandemic.

Mental Health and Behavior of College Students During the Early Phases of the COVID-19 Pandemic: Longitudinal Smartphone and Ecological Momentary Assessment Study.

BACKGROUND: The vast majority of people worldwide have been impacted by coronavirus disease (COVID-19). In addition to the millions of individuals who have been infected with the disease, billions of individuals have been asked or required by local and national governments to change their behavioral patterns. Previous research on epidemics or traumatic events suggests that this can lead to profound behavioral and mental health changes; however, researchers are rarely able to track these changes with frequent, near-real-time sampling or compare their findings to previous years of data for the same individuals. OBJECTIVE: By combining mobile phone sensing and self-reported mental health data among college students who have been participating in a longitudinal study for the past 2 years, we sought to answer two overarching questions. First, have the behaviors and mental health of the participants changed in response to the COVID-19 pandemic compared to previous time periods? Second, are these behavior and mental health changes associated with the relative news coverage of COVID-19 in the US media? METHODS: Behaviors such as the number of locations visited, distance traveled, duration of phone usage, number of phone unlocks, sleep duration, and sedentary time were measured using the StudentLife smartphone sensing app. Depression and anxiety were assessed using weekly self-reported ecological momentary assessments of the Patient Health Questionnaire-4. The participants were 217 undergraduate students, with 178 (82.0%) students providing data during the Winter 2020 term. Differences in behaviors and self-reported mental health collected during the Winter 2020 term compared to previous terms in the same cohort were modeled using mixed linear models. RESULTS: During the first academic term impacted by COVID-19 (Winter 2020), individuals were more sedentary and reported increased anxiety and depression symptoms (P<.001) relative to previous academic terms and subsequent academic breaks. Interactions between the Winter 2020 term and the week of the academic term (linear and quadratic) were significant. In a mixed linear model, phone usage, number of locations visited, and week of the term were strongly associated with increased amount of COVID-19-related news. When mental health metrics (eg, depression and anxiety) were added to the previous measures (week of term, number of locations visited, and phone usage), both anxiety (P<.001) and depression (P=.03) were significantly associated with COVID-19-related news. CONCLUSIONS: Compared with prior academic terms, individuals in the Winter 2020 term were more sedentary, anxious, and depressed. A wide variety of behaviors, including increased phone usage, decreased physical activity, and fewer locations visited, were associated with fluctuations in COVID-19 news reporting. While this large-scale shift in mental health and behavior is unsurprising, its characterization is particularly important to help guide the development of methods to reduce the impact of future catastrophic events on the mental health of the population.